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ABSTRACT 

In laboratory courses, the assessment of exercises and assignments typically is treated as a simple, quantifiable approach. 
This approach however rarely includes qualitative factors, especially if the grading is being automatically performed by 
the system, and provides little to no feedback for the students to reflect on their work. The role of the laboratory in 
engineering education however is very critical and engineering students must possess knowledge that goes beyond mere 
theory, therefore a diverse and multidimensional assessment of laboratory students is a necessity. In this paper, an 
educational tool for laboratory students implementing automatic assessment based on artificial intelligence is being 
proposed, based on a pilot version that has been developed for MATLAB -related coursework. The highlights of the 
proposed tool is that it is based on a proven cognitive theory, it is easy to compile and or modify its contents, it is based 
on the same laboratory environment that the students are being trained in, it can include qualitative evaluation and it can 
save time that the educators would otherwise require to manually evaluate the exercises. 
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1. INTRODUCTION 

In laboratory courses, the assessment of exercises and assignments typically is treated as a simple, 
quantifiable approach. The student turns in an assignment and or exercise and the educator or system is 
expected to grade it. This approach however rarely includes qualitative factors, especially if the grading is 
being automatically performed by the system, and provides little to no feedback for the students to reflect on 
their work. This is especially true if the assessment is based on simple questions that have only one correct 
answer (Bush 1999), or if the assessment is only based on the end result, neglecting the efficiency of the 
solution (Carter et al. 2003). The role of the laboratory in engineering education however is very critical and 
engineering students must possess knowledge that goes beyond mere theory (Feisel and Rosa 2005), 
therefore a diverse and multidimensional assessment of laboratory students is a necessity. 

Educators can examine and grade the exercises either manually or by using automated tools. Automated 
tools significantly reduce the effort required for such a task, especially when there are a significant number of 
students; however, they are usually limited to a quantitative evaluation and cannot provide specialized 
feedback to the students (Douce et al. 2005). Depending on the laboratory course, they also may operate 
outside the actual tool/software that the students are being educated on, in an educational environment alien 
to the students, frequently treating the examination the same way as that of a theoretical subject (Ihantola et 
al. 2010). 

In this paper, we suggest the creation of an automatic laboratory course assessment tool that can provide 
personalized feedback to each student, allowing the students to discern their strength and weaknesses. The 
approach suggests that the tool should operate in the same environment that students have been using during 
their laboratory course. For this purpose, we chose to present an adapted MATLAB course, as it is the most 
widely used software in laboratory engineering education and covers a wide range of subjects (Harris et al. 
2002, Schmid and Ali 2000, de Magistris 2005). The approach presented in this paper is based on the text 
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comprehension theory of Denhiere & Baudet (Denhiere and Baudet 1992). This suggests that the texts and 
assessment questions of the tool should be adjusted into three different versions, R (Relational), M 
(Transformative) and T (Teleological). Details on the adaptation of material to meet the requirements of the 
applied comprehension theory can be found in (Samarakou et al. 2013b, Samarakou et al. 2013c). 


2. DEVELOPMENT OF THE AUTOMATIC ASSESSMENT TOOL 

The main objective of the content management system should be the easy, quick and user-friendly handling 
of both the educational resources and student data, in order to require little or no experience for anyone with 
basic computer skills to use it. As such, it is vital to simplify the system as much as possible by implementing 
widely known tools and resources. As MATLAB is a very flexible software, it is easy to develop an 
automated assessment tool that the students may use without having to result into another environment. The 
entire process can be introduced as a set of MATLAB ".m" files, programmed to initiate the assessment 
process once the student decides to access the software. The use of a spreadsheet file as a database for the 
educational material is also recommended; as MATLAB is capable of accessing spreadsheet files, it is very 
easy for the educator to provide the educational material via such a file, which may be very easily updated 
and or modified by virtually everyone with basic computer knowledge. Such a file can include everything 
that is required for the coverage of a specific topic, including the questions, technical texts and the data 
related to the performance of the students (Fig. 1). As such, it is easy for the educator to check the 
performance of each student and or automatically create a summary file with the performance of any number 
of students. 



Figure 1 . Example of a spreadsheet file used as a database for MATLAB 


2.1 Specifications and Requirements 

In order to develop an automatic MATLAB assessment tool based on the proposed cognitive theory, there are 
a number of requirements that need to be addressed. The requirements are both technical and educational and 
can be split into two groups, those intended for educators and those intended for students. 
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For educators, the requirements are: 

1 . The ability to create a virtually infinite number of new educational material 

2. The ability to input/modify new educational material (texts, questions, etc.) into suitable fields, 
without restrictions on their number, as well as the number of characters and answers to each 
question. 

3. The ability to designate the kind of text/question entered into the database, according to the cognitive 
theory. 

4. The ability to create activities with appropriate connection to the inserted educational material. 

5. The ability to create diagnostic tests for a single section with a selection, random or forced, from the 
pre -inserted questions 

6. Protection of the database file, so that it will not be modified by unauthorized users. 

7. The ability to share educational material between authorized authors/users. 

For students, the requirements are: 

1. The ability to register into the system (Name, Surname, ID Number) 

2. User-friendly interface 

3. The ability to choose any path after the diagnostic test, regardless of the received suggestion 

4. The short and easy explanation of the differences between the different texts and activities offered 
by the system (R, M and T type in our particular example) 

5. The ability to request additional assistance/feedback regarding a question and or text 

6. The ability to see the results of the taken tests 

2.2 The Student Environment 

Once the educator has set up the appropriate files, they can be either inserted into the workstations of a 
laboratory before the examination or provided to the students via an online portal. If the educational process 
is taking place into a controllable environment, such as a laboratory, the system can be programmed to 
deliver and or combine the assessment results of each student to the computer of the educator through the 
laboratory's computer network. If the files are given through an online portal, the files with the results will 
have to be sent manually back to the educator and they should be followed by appropriate instructions. It is 
thus vital for the file containing the results to be protected against unauthorized access, to prevent tampering 
before it is sent back to the educator. 

The MATLAB .m files should be programmed to be able to utilize a simple content database, which is a 
typical spreadsheet file in our example, and generate a file with the assessment results of each student. In 
order to do that, the program will ask the student of some basic identification data, such as the name, surname 
and student ID number. Based on the comprehension theory implemented for our example, there should first 
be a diagnostic test, assessing the prior knowledge of the student, followed by a recommendation to access 
further educational material. This educational material may be of either R, M or T type, depending on the 
cognitive profile of the student generated from the results of the first test (Samarakou et al. 2013c). 

Once the diagnostic test is completed, the student is provided feedback on particular strengths and 
weaknesses. The software will automatically assess on which type of cognition the student is the least 
competent on (R, M or T) and will offer feedback to the student, alongside with the suggestion to supply 
additional educational resources. It is vital to allow the student to choose resources of any type, regarding the 
results of the text, or even skip the additional material entirely and proceed to the final test. After the student 
completes the final test, the system should generate a file with the student's answers, profile, results and other 
data of relevance that may be recorded during the test, such as the time taken to answer each question and the 
number of times the student requested additional assistance from the software. Such data may be used to add 
a qualitative portion into the evaluation process (Samarakou et al. 2013a). After the completion of the 
educational process, the student should be clearly informed of his/her progress by the system in detail, which 
includes his/her detailed performance on every (diagnostic, feedback, final) stage of the process. 

The students should be using MATLAB during the entire process, working in the same environment that 
they were being taught during the course. The tool should provide questions and problems by automatically 
and possibly randomly extracting them from the content database file. These questions can range from typical 
"True - False" type to simple problems. It is also possible to add the ability for the student to request 
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additional help during the process. In our example, this ability has been implemented, as it is a recommended 
part of the cognitive theory that the proposed system is based on, and the student can request additional help 
at any time by simply typing a command. 

2.3 Evaluation Results 

A preliminary evaluation of the proposed system took place in the Informatics laboratory of the 
Technological Educational Institute of Athens, during the course "Introduction to MATLAB". A group of 40 
students has been asked to use a preliminary version of the tool and then provide feedback through a short 
questionnaire. The students are expected to take at least ten laboratory classes per course, accompanied by 
ten laboratory tests. The laboratory tests are normally given and assessed manually by the educators, but one 
test has been skipped and the students have been asked to use the proposed tool instead. The questionnaire 
and the results of the survey are displayed in Table 1. 


Table 1 . Survey Feedback 


Question 

Strongly 

Agree 

Agree 

Neither 
Agree 
or Disagree 

Disagree 

Strongly 

Disagree 

The tool was straightforward and easy to use 

80% 

15% 

5% 

0% 

0% 

The instructions were clear and helpful 

70% 

12.5% 

15% 

2.5% 

0% 

The tool has helped to improve your comprehension 
on the subject 

75% 

15% 

2.5% 

5% 

2.5% 

The time required to complete the process was 
reasonable 

40% 

40% 

15% 

2.5% 

2.5% 

You believe that the automatic assessment via the 
tool was fair 

50% 

35% 

20% 

5% 

0% 

You would like the tool to replace all standard 
laboratory tests in the future 

5% 

40% 

30% 

20% 

5% 


As it can be seen from the results of the survey, the students reacted very positively and generally enjoyed 
using the proposed tool over their typical manual assessment. As the students were given the exact same time 
to complete the tests of the tool as their common manual tests, most found the tool faster and easier to use 
than the manual test, despite the fact that additional educational material was being given to them during the 
process, and they generally found the assessment process fair and transparent. When asked if they would like 
all of their manual tests to be replaced by the automatic assessment tool, the results were generally favorable 
but more diverse than those of the previous five questions; this indicates that a slower adoption process 
should take place, in order to both resolve any problems prior to completely automating the process and to 
quell the worries of the students. 


3. CONCLUSION 

In this paper, an automated evaluation tool for the laboratory training of engineering students has been 
proposed. The proposed tool has been based on the well-known text comprehension theory of Denhiere & 
Baudet, forming a cognitive profile of the student via a diagnostic test and offering personalized feedback 
and assistance based on the results of this test. This helps to overcome the "one size fits all" approach, which 
is dominant in engineering education today. Similar tools can also be developed based on other 
comprehension theories as well. 

The system proposed in this study has multiple advantages. From an educational point of view, the 
greatest advantage is that the student continues to use the same tool he has been receiving training on, 
without having to result into a classic "pen and paper" type of test or an alien eLearning environment. For 
educators, the advantages of the proposed approach are that it is simple and effective. By making use of 
typical computer files, such as spreadsheets, the modification and or addition of new educational resources is 
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a simple and straightforward matter for anyone with basic computer skills. The time that it would require to 
develop such a tool should be less than the time required to manually assess the exercises of a single course. 

Assessment via the proposed tool can be multidimensional, implementing both quantitative and 
qualitative factors. Depending on the cognitive model used, aside from the number of correct, wrong and 
unanswered questions, qualitative factors can also be implemented into the assessment. These can be the 
choice of the additional educational material, if the user skipped the additional material entirely and 
proceeded to the test, the time required to complete each question and the entire test, the number of times that 
the user requested assistance, etc. 

The proposed tool can also be used for online and distance training, provided that the student will be 
instructed to return the file with the results to the educator or informed that the file will be automatically sent 
to the educator once the test has been completed. The only drawback of such an approach is that the 
computer of the user must have a licensed version of the software tool used for the educational process, as the 
system is based on the actual tool itself. 
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